Objective: The purpose of the project is to design and develop the necessary algorithms for a full sperm-test analysis. We first develop an android application for capturing a video of the sperm sample so it can be analysed by computer vision algorithms. To decouple the acquisition and analysis steps and also facilitate in fast execution of the algorithms, we implement a server-client communication model, so that after acquisition, the video is transferred to a central server. Upon processing, the results are sent back to the mobile device where they are presented to the user. Finally, insight on extending the project into a fully functional hardware device is also provided here. Upon integration with the hardware, the device would potentially offer a point-of-care cost-affordable alternative for the costly, and often uncomfortable for the patient, procedure of laboratory sperm examinations.
Definition The spermodiagram is a medical examination which tests the health of a patient's semen, and helps find the cause of problems in procreation by measuring the motility, the number, the vitality and the pH of the subject’s sperm. Examination procedureBefore a sample is given, the patient should be sexually inactive for 2-5 days. After giving a sample, the semen is kept at body temperature for 20-60 minutes so it can liquefy. Afterward, the doctor pours a drop of the sample onto a microscope slide with a pipette and seals it with a cover slip. He then starts adjusting the lenses of the microscope so that he could have the best possible image to evaluate the sample in the best possible way. The term ‘best’ is quite relevant. The best image is defined by the surface of the sample that will include enough sperms to evaluate and in this surface the sperm that is examined will not move away from the examined frame.Then the doctor calculates the sperm this is achieved with a Neubauer plate. The "lenses" theoryRegarding the lenses, it was observed that an appropriate image is achieved at 20X10, and 40X10 zoom. These magnifications can be achieved with two lenses, one objective and one eyepiece. The objective lens is responsible for the magnification of the sample and the eyepiece lens for the zooming of the image. The difference is that the first kind of lenses "adds more information" to the image, while the other kind just makes the image bigger. Similar ProjectsA research was conducted in order to find projects similar to this one. There were only few that had the same objective and are presented below: I-Sperm: is a very similar project to ours but it is applied for animal examination. The concept is simple. I-sperm is a structure consisting of a number of lenses and a specially designed basis on which the sperm is based. This devise is mounted on an i-pad's camera and then a video is taken. The program starts running and after a couple of seconds you know if your animals are fertile. There are two important drawbacks in this project as the software is only compatible with i-pads and moreover the device is too expensive. A single i-sperm costs approximately 1500 $. Virus and Nano-Particle Imager: device created by the company "Holonomic". It weights less than 200 grams and detects single 100 nm. The imaging results of a smart phone platform were validated using scanning electron microscopy images of the same sample.
In the above images, the left one is the I-sperm device and the right one the "Virus and Nano-Particle Imager"
The rest of the projects that were found were based on the same principle. In General, they did not count exclusively measure the sperm condition. The structures of these projects were simple with the lens either applied on the phone’s camera directly or supported by a small basis that can be applied on a mobile phone. The problems with these projects were either that the necessary magnification to conduct sperm analysis could not be achieved or the overall structure of the device was exceeding the dimensions of a typical pharmaceutical product. The team has managed to find ways to deal with the aforementioned problems, as they are described below.
General Description - Using the device
A typical usage of the device consists of the following steps.
Purchase of the device from a pharmacy. A set of additional material will be packed together so that the overall execution can be automated
(2) microscope slides
(2) Cover slips
(2) Pipettes
Device setup.
The user should deposit a small amount of the sperm sample between a slide and a slip and place it in the designated position in the device.
Sperm sample analysis and results
Afterwards the user should place his phone on top of the device in a horizontal orientation and launch the Sperm3000 application (downloaded from the Android Market). Then, following the on-screen instructions, a fixed-length video will be taken and analysed. Finally the results are printed in the user's screen.
Specifications and Goals
The main goals set for the software under-development can be summarised as follows:
Compatibility w/ the majority of phones in the market.
Fast and accurate results independent of the user's smartphone
These abstract goals can be further broken down into the following low-level technical specifications:
Android application designed to work on the majority of Android platforms currently in the market
Design for Android API > 15
Setup of an external server node. The smartphone application connects and sends the data so that the analysis can proceed in the same speed and independent of the running android platform and its computational capabilities
Conceptual Design Brainstorming
Following, are the two main alternatives to the designed that we finally came up with.
The first one was influenced by one of the devices that already exists and is presented in Figure 1.
Figure 1: First idea about the device. Although, it is a lightweight design and it can reach the desired magnification it has a couple of disadvantages:
Due to its large volume it would be difficult to be shipped or sold in a pharmacy in large amounts. Even if it was disassembled into parts, it would still not be suitable for its purpose.
In order to reach the requested magnification we had to make use of the digital zoom of the smartphone's camera. So, we would lose quality
A second alternative is to use a device with a rectangular cross-section that could lie in any flat surface. The smartphone would be placed on the top of the device and two mirrors would be used so that the overall height would decrease. The first would place a virtual image in parallel to the flat surface axis and the other one would be used after the lenses to form a virtual image vertical to the flat surface axis. However, excessive usage of mirrors would render the design was too complicated without offering any significant advantage in comparison to the design we finally came up with and is shown in Figure 2.
Figure 2: Final idea about the device.
Note: A thorough analysis of the final design is given in milestone 2: (Detailed design>Structural Part>Device design).
Milestone 2: Detailed design
Based on the previous analysis the project can be broken down into two major parts:
Software part
Structural
We concentrate our analysis on the software part and give an insight into a potential implementation of the structural in the final paragraph of the page.
Software
The software part comprises three subparts which are listed and explained in detail below:
Image analysis part
Android application
Server side communication
Image analysis
Based on the specifications set, the main objectives of the algorithm are to compute an estimation of the total number of spermatozoa in a given image (given the overall volume in a user's sample) as well as motility stats of the detected spermatozoa. Due to its versatility and ease in writing boilerplate code, the initial version of the image analysis algorithm which is to run on the server is coded in MATLAB. The algorithm which was built has these main functions:
Read a video
Learn the background (non-moving objects over the entire sequence of frames) and the foreground (moving spermatozoa)
Apply morphological operations in order to recognize the moving objects in the video and filter out the image noise
Count the moving objects in each frame and find the average
Track the moving objects and calculate their velocity
In more detail, Matlab functions special for image processing were used for our purpose. The most significant of them are presented below:
vision.ForegroundDetector (System object) DescriptionThe ForegroundDetector System object compares a color or grayscale video frame to a background model to determine whether individual pixels are part of the background or the foreground. It then computes a foreground mask. By using background subtraction, you can detect foreground objects in an image taken from a stationary camera. Constructiondetector = vision.ForegroundDetector returns a foreground detector System object, detector. Given a series of either grayscale or color video frames, the object computes and returns the foreground mask using Gaussian mixture models (GMM).PropertiesSome of the properties with values that affect the way this function is applied are:NumTrainingFrames (specifies the number of initial video frames for training background model), MinimumBackgroundRatio, (threshold to determine background model), NumGaussians (specifies the number of Gaussian modes in the mixture model).
vision.BlobAnalysis (System object) DescriptionThe BlobAnalysis object computes statistics for connected regions in a binary image.Using the step syntax with input binary image, blob analysis object and any optional properties will return statistics of the input binary image depending on the property values specified.ConstructionH = vision.BlobAnalysis returns a blob analysis System object, H, used to compute statistics for connected regions in a binary image.H = vision.BlobAnalysis(Name,Value) returns a blob analysis object, H, with each specified property set to the specified value.PropertiesSome of the properties of this function are AreaOutputPort, CentroidOutputPort, BoundingBoxOutputPort, which are properties responsible for the outputs, as long as with properties such as Connectivity, MaximumCount, MaximumBlobArea, which values can be changed in order to service the application.
vision.KalmanFilter (Class) DescriptionThe Kalman filter object is designed for tracking. You can use it to predict a physical object's future location, to reduce noise in the detected location, or to help associate multiple physical objects with their corresponding tracks. A Kalman filter object can be configured for each physical object for multiple object tracking. To use the Kalman filter, the object must be moving at constant velocity or constant acceleration.The Kalman filter algorithm involves two steps, prediction and correction (also known as the update step). The first step uses previous states to predict the current state. The second step uses the current measurement, such as object location, to correct the state. The Kalman filter implements a discrete time, linear State-Space System.Constructionobj = vision.KalmanFilter returns a Kalman filter object for a discrete time, constant velocity system.To track objects, the predict and correct methods must be used based on detection results.This object implements a discrete time, linear state-space system, described by the following equations:
State prediction:
x(k)=Ax(k−1)+Bu(k−1)+w(k−1)
Measurement update:
z(k)=Hx(k)+v(k)
configureKalmanFilter
Description
kalmanFilter = configureKalmanFilter (MotionModel, InitialLocation, InitialEstimateError, MotionNoise, MeasurementNoise)returns a vision.KalmanFilter object configured to track a physical object. This object moves with constant velocity or constant acceleration in an M-dimensional Cartesian space. The function determines the number of dimensions, M, from the length of the InitialLocation vector.
This function provides a simple approach for configuring the vision.KalmanFilter object for tracking a physical object in a Cartesian coordinate system. The tracked object may move with either constant velocity or constant acceleration. The statistics are the same along all dimensions.The Kalman filter implements a discrete time, linear State-Space System. The configureKalmanFilter sets the vision.KalmanFilter object properties.
In the videos below, one can see the results of the sperm concentration algorithm:
Image processing for calculating the concentration of spermatozoa in a sequence of frames. The algorithm detects foreground moving objects in a binary image and plots a rectangle in the initial video for the spermatozoa that are detected
In order to test the behavior of the tracking algorithm we plotted the trajectory of two spermatozoa during a sequence of the given video. The results are given in the following images: Fig. 3-4. Plots of the positions of random spermatozoa through the frames. The positions were given as outputs of configureKalmanFilter. Having the average number of spermatozoa (moving objects) that are seen in a frame, it is possible to calculate the total concentration, based on the total volume of the sample, the volume of sample that was put on the slide (parameters that the user has to input in the beginning), the area of the sample on the slide and the area of what is seen with the camera. The calculation of the latter requires calibration of camera used, based on the video that is produced by the application. For the development of this algorithm, as the device is not yet ready, the videos that were used were taken from the microscope of the laboratory. For these videos (with resolution 1024x768), camera calibration gave the following results:
For magnification 10x: 472x354 μm (1 pixel equals to 0.461 μm)
For magnification 20x: 231x174 μm (1 pixel equals to 0.226 μm)
For magnification 40x: 165x124 μm (1 pixel equals to 0.161 μm)
Moreover, with the result of the operation of tracking the spermatozoa (moving objects), it is possible to calculate the velocity of each track, through the position that had in every frame. The direct calculation can give pixels/frame. To convert this to μm/sec, what is needed is the frame frequency of the video (which in this case is 15 frames/sec) and, from camera calibration the amount of micrometers that are presented by one pixel, as seen above. After this, the algorithm classify the tracks in categories based on their velocity and gives as output the percentage of spermatozoa that belong in each one. The categories are the following:
Fast forward movement A > 25 μm/sec
Slow forward movement B 5 - 25 μm/sec
Movement on the spot C < 5 μm/sec
The code for the image analysis algorithms previously mentioned can be accessed through Github.
Android application - Sperm3000
For capturing video of the sperm sample an android application has been developed. This was decided, given that most of the phones in the market utilise the Android OS, the application and thus the device audience increases dramatically. The application is heavily based on the Open Camera Project . This decision was due to the camera setting capabilities and features that Open Camera offers (auto-stabilise, fix-length of video, timers, Camera2 API support etc).
Furthermore in order to run on most modern android devices the application is coded having API 15 as the minimum API that can run on. This way the application can run on all android versions, since Android 4.0 (Ice Cream Sandwich) which add up to 94% of the total number of Android devices currently in the market.
Application InterfaceThe application consists of the following screens:
Initial/Homescreen
From which the user states his intention (Shoot video / see previous results etc)
CameraThe user shoots a predetermined fixed-length video.
SettingsTinkering with application settings as well as setting another IP/port combination is available.
A typical preview of the homescreen and camera screens is given below:
Fig. 5. Application's homescreen and camera screen
Communication with server
In order to make the analysis of the video independent of the computational capabilities of the user's, we implemented a client-socket communication protocol so that the device can send the video to an external server and then wait for the results and print them accordingly to the screen of the user. A TCP Socket client is initialized from within the application and connects to a predefined IP/Port pair which corresponds to the analysis server. Before sending the data to the server, the java client encodes the data in base64 binary format. In order to make the communication as robust as possible, the client is coded using a Client Thread so that the application will not block every time the connection is set up. A sample of the ClientThread code is given below:
The application was exclusively coded in Java, using the Google Android Studio IDE. The code as well as the .APK (which is currently in DEBUG mode) can be found in Github.
Server-side communication
For the video to be transferred to the server, a TCP Socket server is set up using Python. Although it would have been straightforward to setup both the image analysis algorithm and the TCP server in MATLAB, we decided to use Python for all networking purposes and keep MATLAB to the computer vision part exclusively. This was decided having the extensibility of the project in mind, as Python is the standard language for networking and communications in the majority of projects. After the python server successfully accepts the connection, it receives the video and stores it in a file under the directory (/videos_in/pending/). After the transfer has completed, it decodes the base64 binary data and moves the file to the /videos_in/done/ directory. Meanwhile the MATLAB computational node, using the watchDog_dir module, scans the latter directory for changes, and when it detects a new video it uses it as input for the image analysis algorithm. After the sperm analysis is completed MATLAB writes the results in a "Python-Watched" directory (/results_out/done/). Finally python having detected the changes in the aforementioned directory sends the updated content (a results textfile) back to the client. The process described can be summarised in the following figure.
Figure 6. Flow chart of the communication between software parts
A sample of the watchDogDir function is given below:
function [changed, changes_in, changes_out] = watchDogDir(dirpath, timeout)
% WATCHDOG_DIR watches if the contents of a specific directory have changed by checking
% its contents. If it does it returns with code 1 as well as two lists of changes
% with regards to the initial contents. User can also specify an optional
% timeout parameter for the maximum time to wait until the function returns
%
% nickkouk, 20150112
% input validation - set the loop condition based on weather there is a
% timeout
if nargin == 2
maxTime = timeInSecs() + timeout;
condition = @(x) maxTime - timeInSecs();
else
condition = @(x) true;
end
fid = 1;
fprintf(fid, 'Waiting for change in directory: %s\n', dirpath);
waitBetweenChecks = 0.5;
initDirConts = dir(dirpath);
initDirLength = length(initDirConts);
initDirNames = {initDirConts.name};
while condition()
curDirConts = (dir(dirpath));
curDirLength = length(curDirConts);
% check against the initial value
if curDirLength ~= initDirLength
changed = true;
curDirNames = {curDirConts.name};
% setdiff - returns the values in A that are not in B!
changes_in = setdiff(curDirNames, initDirNames);
changes_out = setdiff(initDirNames, curDirNames);
return;
end
pause(waitBetweenChecks);
end
% if reached timeout has passed - return false.
changed = false;
changes_in = {};
changes_out = {};
return;
end
A demonstration of the whole communication protocol is presented in the following video. A sample video is taken through the android phone, which is sent to a server. As described below MATLAB is eventually called (but runs with a predetermined video file since we didn't use real semen sample):
The code for the server modules are available in Github.
Structural part
As part of the project we also made a thorough analysis into the mathematics and hardware / physical requirements for building a sperm analysis device. You can find this work as well as a more detailed overall analysis of the project here.
Future thoughts - Extending the project
With regards to the software part, the following features and changes have to be implemented:
Since the android application is based on Open Camera, and since Open Camera is licensed under GPL v3, no commercial product can come out of it. Therefore, the camera part of the application has to be rewritten and licensed under a different license (e.g MIT, 2-Clause BSD etc).
The video has to be delivered to the server in an encrypted manner. As an improvement in the communication protocol, the client can first encrypt the data using standard encryption techniques (AES 128, 256 etc).
With regards to the image processing part, the code execution can be optimised using the following methods
Use of a compiled lower-level language. C++ is a realistic choice due to its object-oriented nature as well as the variety of image-analysis codes available.
Send and process video at the same time, in a live-stream fashion, so the overall computation time of the results is decreased.
Acknowledgements
We would like to thank the following people for their indispensable support and valuable insight during this project :
Project Info
Objective:The purpose of the project is to design and develop the necessary algorithms for a full sperm-test analysis.
We first develop an android application for capturing a video of the sperm sample so it can be analysed by computer vision algorithms. To decouple the acquisition and analysis steps and also facilitate in fast execution of the algorithms, we implement a server-client communication model, so that after acquisition, the video is transferred to a central server. Upon processing, the results are sent back to the mobile device where they are presented to the user.
Finally, insight on extending the project into a fully functional hardware device is also provided here. Upon integration with the hardware, the device would potentially offer a point-of-care cost-affordable alternative for the costly, and often uncomfortable for the patient, procedure of laboratory sperm examinations.
Team Members:
1. Mavroudis Angelos
2. Parousi - Papadopoulou Athina
3. Koukis Nikos
4. Derventzis Konstantinos
Team Contact:
Koukis Nikos - nickkouk@gmail.com
Mentor:
Alexopoulos Leonidas (leo@mail.ntua.gr)
Consultants:
Kanakaris Georgios (gkanak84@gmail.com)
Milestone 1: Preliminary design
Introduction
DefinitionThe spermodiagram is a medical examination which tests the health of a patient's semen, and helps find the cause of problems in procreation by measuring the motility, the number, the vitality and the pH of the subject’s sperm.
Examination procedureBefore a sample is given, the patient should be sexually inactive for 2-5 days. After giving a sample, the semen is kept at body temperature for 20-60 minutes so it can liquefy. Afterward, the doctor pours a drop of the sample onto a microscope slide with a pipette and seals it with a cover slip. He then starts adjusting the lenses of the microscope so that he could have the best possible image to evaluate the sample in the best possible way. The term ‘best’ is quite relevant. The best image is defined by the surface of the sample that will include enough sperms to evaluate and in this surface the sperm that is examined will not move away from the examined frame.Then the doctor calculates the sperm this is achieved with a Neubauer plate.
The "lenses" theoryRegarding the lenses, it was observed that an appropriate image is achieved at 20X10, and 40X10 zoom. These magnifications can be achieved with two lenses, one objective and one eyepiece. The objective lens is responsible for the magnification of the sample and the eyepiece lens for the zooming of the image. The difference is that the first kind of lenses "adds more information" to the image, while the other kind just makes the image bigger.
Similar ProjectsA research was conducted in order to find projects similar to this one. There were only few that had the same objective and are presented below:
I-Sperm: is a very similar project to ours but it is applied for animal examination. The concept is simple. I-sperm is a structure consisting of a number of lenses and a specially designed basis on which the sperm is based. This devise is mounted on an i-pad's camera and then a video is taken. The program starts running and after a couple of seconds you know if your animals are fertile. There are two important drawbacks in this project as the software is only compatible with i-pads and moreover the device is too expensive. A single i-sperm costs approximately 1500 $.
Virus and Nano-Particle Imager: device created by the company "Holonomic". It weights less than 200 grams and detects single 100 nm. The imaging results of a smart phone platform were validated using scanning electron microscopy images of the same sample.
In the above images, the left one is the I-sperm device and the right one the "Virus and Nano-Particle Imager"
The rest of the projects that were found were based on the same principle. In General, they did not count exclusively measure the sperm condition. The structures of these projects were simple with the lens either applied on the phone’s camera directly or supported by a small basis that can be applied on a mobile phone. The problems with these projects were either that the necessary magnification to conduct sperm analysis could not be achieved or the overall structure of the device was exceeding the dimensions of a typical pharmaceutical product. The team has managed to find ways to deal with the aforementioned problems, as they are described below.
General Description - Using the device
A typical usage of the device consists of the following steps.(2) microscope slides
(2) Cover slips
(2) Pipettes
The user should deposit a small amount of the sperm sample between a slide and a slip and place it in the designated position in the device.
Afterwards the user should place his phone on top of the device in a horizontal orientation and launch the Sperm3000 application (downloaded from the Android Market). Then, following the on-screen instructions, a fixed-length video will be taken and analysed. Finally the results are printed in the user's screen.
Specifications and Goals
The main goals set for the software under-development can be summarised as follows:
- Compatibility w/ the majority of phones in the market.
- Fast and accurate results independent of the user's smartphone
These abstract goals can be further broken down into the following low-level technical specifications:Design for Android API > 15
Conceptual Design Brainstorming
Following, are the two main alternatives to the designed that we finally came up with.
The first one was influenced by one of the devices that already exists and is presented in Figure 1.
Although, it is a lightweight design and it can reach the desired magnification it has a couple of disadvantages:
- Due to its large volume it would be difficult to be shipped or sold in a pharmacy in large amounts. Even if it was disassembled into parts, it would still not be suitable for its purpose.
- In order to reach the requested magnification we had to make use of the digital zoom of the smartphone's camera. So, we would lose quality
A second alternative is to use a device with a rectangular cross-section that could lie in any flat surface. The smartphone would be placed on the top of the device and two mirrors would be used so that the overall height would decrease. The first would place a virtual image in parallel to the flat surface axis and the other one would be used after the lenses to form a virtual image vertical to the flat surface axis. However, excessive usage of mirrors would render the design was too complicated without offering any significant advantage in comparison to the design we finally came up with and is shown in Figure 2.Note: A thorough analysis of the final design is given in milestone 2: (Detailed design>Structural Part>Device design).
Milestone 2: Detailed design
Based on the previous analysis the project can be broken down into two major parts:
We concentrate our analysis on the software part and give an insight into a potential implementation of the structural in the final paragraph of the page.
Software
The software part comprises three subparts which are listed and explained in detail below:Image analysis
Based on the specifications set, the main objectives of the algorithm are to compute an estimation of the total number of spermatozoa in a given image (given the overall volume in a user's sample) as well as motility stats of the detected spermatozoa. Due to its versatility and ease in writing boilerplate code, the initial version of the image analysis algorithm which is to run on the server is coded in MATLAB.
The algorithm which was built has these main functions:
In more detail, Matlab functions special for image processing were used for our purpose. The most significant of them are presented below:
vision.ForegroundDetector (System object)
DescriptionThe ForegroundDetector System object compares a color or grayscale video frame to a background model to determine whether individual pixels are part of the background or the foreground. It then computes a foreground mask. By using background subtraction, you can detect foreground objects in an image taken from a stationary camera.
Constructiondetector = vision.ForegroundDetector returns a foreground detector System object, detector. Given a series of either grayscale or color video frames, the object computes and returns the foreground mask using Gaussian mixture models (GMM).PropertiesSome of the properties with values that affect the way this function is applied are:NumTrainingFrames (specifies the number of initial video frames for training background model), MinimumBackgroundRatio, (threshold to determine background model), NumGaussians (specifies the number of Gaussian modes in the mixture model).
vision.BlobAnalysis (System object)
DescriptionThe BlobAnalysis object computes statistics for connected regions in a binary image.Using the step syntax with input binary image, blob analysis object and any optional properties will return statistics of the input binary image depending on the property values specified.ConstructionH = vision.BlobAnalysis returns a blob analysis System object, H, used to compute statistics for connected regions in a binary image.H = vision.BlobAnalysis(Name,Value) returns a blob analysis object, H, with each specified property set to the specified value.PropertiesSome of the properties of this function are AreaOutputPort, CentroidOutputPort, BoundingBoxOutputPort, which are properties responsible for the outputs, as long as with properties such as Connectivity, MaximumCount, MaximumBlobArea, which values can be changed in order to service the application.
vision.KalmanFilter (Class)
DescriptionThe Kalman filter object is designed for tracking. You can use it to predict a physical object's future location, to reduce noise in the detected location, or to help associate multiple physical objects with their corresponding tracks. A Kalman filter object can be configured for each physical object for multiple object tracking. To use the Kalman filter, the object must be moving at constant velocity or constant acceleration.The Kalman filter algorithm involves two steps, prediction and correction (also known as the update step). The first step uses previous states to predict the current state. The second step uses the current measurement, such as object location, to correct the state. The Kalman filter implements a discrete time, linear State-Space System.Constructionobj = vision.KalmanFilter returns a Kalman filter object for a discrete time, constant velocity system.To track objects, the predict and correct methods must be used based on detection results.This object implements a discrete time, linear state-space system, described by the following equations:
configureKalmanFilter
Description
kalmanFilter = configureKalmanFilter (MotionModel, InitialLocation, InitialEstimateError, MotionNoise, MeasurementNoise)returns a vision.KalmanFilter object configured to track a physical object. This object moves with constant velocity or constant acceleration in an M-dimensional Cartesian space. The function determines the number of dimensions, M, from the length of the InitialLocation vector.
This function provides a simple approach for configuring the vision.KalmanFilter object for tracking a physical object in a Cartesian coordinate system. The tracked object may move with either constant velocity or constant acceleration. The statistics are the same along all dimensions.The Kalman filter implements a discrete time, linear State-Space System. The configureKalmanFilter sets the vision.KalmanFilter object properties.
(For more information about the above functions, you can visit http://www.mathworks.com.)
In the videos below, one can see the results of the sperm concentration algorithm:
Image processing for calculating the concentration of spermatozoa in a sequence of frames. The algorithm detects foreground moving objects in a binary image and plots a rectangle in the initial video for the spermatozoa that are detected
In order to test the behavior of the tracking algorithm we plotted the trajectory of two spermatozoa during a sequence of the given video. The results are given in the following images:
Fig. 3-4. Plots of the positions of random spermatozoa through the frames. The positions were given as outputs of configureKalmanFilter.
Having the average number of spermatozoa (moving objects) that are seen in a frame, it is possible to calculate the total concentration, based on the total volume of the sample, the volume of sample that was put on the slide (parameters that the user has to input in the beginning), the area of the sample on the slide and the area of what is seen with the camera. The calculation of the latter requires calibration of camera used, based on the video that is produced by the application. For the development of this algorithm, as the device is not yet ready, the videos that were used were taken from the microscope of the laboratory. For these videos (with resolution 1024x768), camera calibration gave the following results:
Moreover, with the result of the operation of tracking the spermatozoa (moving objects), it is possible to calculate the velocity of each track, through the position that had in every frame. The direct calculation can give pixels/frame. To convert this to μm/sec, what is needed is the frame frequency of the video (which in this case is 15 frames/sec) and, from camera calibration the amount of micrometers that are presented by one pixel, as seen above. After this, the algorithm classify the tracks in categories based on their velocity and gives as output the percentage of spermatozoa that belong in each one. The categories are the following:
The code for the image analysis algorithms previously mentioned can be accessed through Github.
Android application - Sperm3000
For capturing video of the sperm sample an android application has been developed. This was decided, given that most of the phones in the market utilise the Android OS, the application and thus the device audience increases dramatically. The application is heavily based on the Open Camera Project . This decision was due to the camera setting capabilities and features that Open Camera offers (auto-stabilise, fix-length of video, timers, Camera2 API support etc).
Furthermore in order to run on most modern android devices the application is coded having API 15 as the minimum API that can run on. This way the application can run on all android versions, since Android 4.0 (Ice Cream Sandwich) which add up to 94% of the total number of Android devices currently in the market.
Application InterfaceThe application consists of the following screens:
From which the user states his intention (Shoot video / see previous results etc)
A typical preview of the homescreen and camera screens is given below:
Communication with server
In order to make the analysis of the video independent of the computational capabilities of the user's, we implemented a client-socket communication protocol so that the device can send the video to an external server and then wait for the results and print them accordingly to the screen of the user. A TCP Socket client is initialized from within the application and connects to a predefined IP/Port pair which corresponds to the analysis server. Before sending the data to the server, the java client encodes the data in base64 binary format. In order to make the communication as robust as possible, the client is coded using a Client Thread so that the application will not block every time the connection is set up. A sample of the ClientThread code is given below:
The application was exclusively coded in Java, using the Google Android Studio IDE. The code as well as the .APK (which is currently in DEBUG mode) can be found in Github.
Server-side communication
For the video to be transferred to the server, a TCP Socket server is set up using Python. Although it would have been straightforward to setup both the image analysis algorithm and the TCP server in MATLAB, we decided to use Python for all networking purposes and keep MATLAB to the computer vision part exclusively. This was decided having the extensibility of the project in mind, as Python is the standard language for networking and communications in the majority of projects. After the python server successfully accepts the connection, it receives the video and stores it in a file under the directory (/videos_in/pending/). After the transfer has completed, it decodes the base64 binary data and moves the file to the /videos_in/done/ directory. Meanwhile the MATLAB computational node, using the watchDog_dir module, scans the latter directory for changes, and when it detects a new video it uses it as input for the image analysis algorithm. After the sperm analysis is completed MATLAB writes the results in a "Python-Watched" directory (/results_out/done/). Finally python having detected the changes in the aforementioned directory sends the updated content (a results textfile) back to the client. The process described can be summarised in the following figure.
Figure 6. Flow chart of the communication between software parts
A sample of the watchDogDir function is given below:
function [changed, changes_in, changes_out] = watchDogDir(dirpath, timeout) % WATCHDOG_DIR watches if the contents of a specific directory have changed by checking % its contents. If it does it returns with code 1 as well as two lists of changes % with regards to the initial contents. User can also specify an optional % timeout parameter for the maximum time to wait until the function returns % % nickkouk, 20150112 % input validation - set the loop condition based on weather there is a % timeout if nargin == 2 maxTime = timeInSecs() + timeout; condition = @(x) maxTime - timeInSecs(); else condition = @(x) true; end fid = 1; fprintf(fid, 'Waiting for change in directory: %s\n', dirpath); waitBetweenChecks = 0.5; initDirConts = dir(dirpath); initDirLength = length(initDirConts); initDirNames = {initDirConts.name}; while condition() curDirConts = (dir(dirpath)); curDirLength = length(curDirConts); % check against the initial value if curDirLength ~= initDirLength changed = true; curDirNames = {curDirConts.name}; % setdiff - returns the values in A that are not in B! changes_in = setdiff(curDirNames, initDirNames); changes_out = setdiff(initDirNames, curDirNames); return; end pause(waitBetweenChecks); end % if reached timeout has passed - return false. changed = false; changes_in = {}; changes_out = {}; return; endA demonstration of the whole communication protocol is presented in the following video. A sample video is taken through the android phone, which is sent to a server. As described below MATLAB is eventually called (but runs with a predetermined video file since we didn't use real semen sample):The code for the server modules are available in Github.
Structural part
As part of the project we also made a thorough analysis into the mathematics and hardware / physical requirements for building a sperm analysis device. You can find this work as well as a more detailed overall analysis of the project here.
Future thoughts - Extending the project
With regards to the software part, the following features and changes have to be implemented:
Acknowledgements
We would like to thank the following people for their indispensable support and valuable insight during this project :
References
Articles/Sites
Further Information
Books
(c) 2015 Department of Mechanical Engineering, National Technical University of Athens.
vision.ForegroundDetector System object